K8SPS-409 | Encrypted backups by mayankshah1607 · Pull Request #1243 · percona/percona-server-mysql-operator

mayankshah1607 · 2026-03-18T04:26:26Z

CHANGE DESCRIPTION

Problem:

Backups taken by the operator are stored unencrypted. Users storing backups in shared or external storage (e.g. S3) have no built-in way to protect them at rest.

Solution:

Adds support for creating and restoring encrypted backups using Percona XtraBackup's built-in encryption. Encryption is enabled by referencing a Secret that holds the encryption key. The default algorithm is AES256; it can be changed via xtrabackup container args.

Usage:

Create a Secret containing the encryption key:

key=$(openssl rand -base64 24)
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: my-encryption-key
stringData:
  encryptionKey: $key
EOF

Enable encryption for all backups by referencing the Secret at the cluster level:

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: ps-cluster1
spec:
  backup:
    encryptionKeySecret:
      name: my-encryption-key
      key: encryptionKey

Optionally, set a per-storage key. When set, it takes precedence over .spec.backup.encryptionKeySecret:

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: ps-cluster1
spec:
  backup:
    storages:
      s3-us-west:
        encryptionKeySecret:
          name: my-encryption-key
          key: encryptionKey

When restoring an external backup, specify the key in the restore resource:

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLRestore
metadata:
  name: restore1
spec:
  clusterName: ps-cluster1
  backupSource:
    storage:
      s3-us-west:
        encryptionKeySecret:
          name: my-encryption-key
          key: encryptionKey

Additional xtrabackup encryption settings (algorithm, threads, chunk size) can be tuned through the existing container args:

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: ps-cluster1
spec:
  backup:
    storages:
      s3-us-west:
        containerOptions:
          args:
            xtrabackup:
            - --encrypt=AES192
            - --encrypt-threads=3
            - --encrypt-chunk-size=3

CHECKLIST

Jira

Is the Jira ticket created and referenced properly?
Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

Is an E2E test/test case added for the new feature/change?
Are unit tests added where appropriate?

Config/Logging/Testability

Are all needed new/changed options added to default YAML files?
Are all needed new/changed options added to the Helm Chart?
Did we add proper logging messages for operator actions?
Did we ensure compatibility with the previous version or cluster upgrade process?
Does the change support oldest and newest supported PS version?
Does the change support oldest and newest supported Kubernetes version?

it-percona-cla · 2026-03-18T04:26:31Z

All committers have signed the CLA.

Copilot

Pull request overview

Adds support for encrypted backups/restores by introducing an encryption spec in backup storage configuration, wiring it through the backup sidecar, and updating restore flow to decrypt before prepare/move-back.

Changes:

Add encryption configuration to Backup/Storage specs and propagate it into the sidecar backup request/config.
Implement encryption key handling in the sidecar (read key from Secret, write temp key file, pass --encrypt-* flags to xtrabackup).
Introduce MySQL pod RBAC/ServiceAccount resources to allow in-pod Secret reads, and update CRDs/bundles/examples accordingly.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
pkg/xtrabackup/xtrabackup.go	Pass encryption options into backup jobs; mount secret + set `ENCRYPTION_KEY_FILE` for restore jobs; extend `BackupConfig`.
pkg/mysql/mysql.go	Add RBAC object constructors; conditionally set MySQL StatefulSet serviceAccountName for newer CR versions.
pkg/controller/ps/controller.go	Create Role/RoleBinding/ServiceAccount during DB reconcile for newer CR versions.
cmd/sidecar/main.go	Switch to constructing a backup handler via `backup.NewHandler()`.
cmd/sidecar/handler/handler.go	Remove backup handler factory wrapper.
cmd/sidecar/handler/backup/handler.go	Add handler constructor/init with a controller-runtime k8s client.
cmd/sidecar/handler/backup/create.go	Create temp encryption key file and pass encryption flags to xtrabackup.
build/run-backup.sh	Include `encryption` in the JSON payload sent to the sidecar.
build/run-restore.sh	Decrypt backup contents when `ENCRYPTION_KEY_FILE` is set.
api/v1/perconaservermysql_types.go	Add `EncryptionSpec` to `BackupStorageSpec` and implement Secret key reading helper.
api/v1/perconaservermysqlbackup_types.go	Add per-backup encryption override + storage fallback getter.
api/v1/zz_generated.deepcopy.go	Update deep-copy generation for new encryption types/fields.
config/crd/bases/*.yaml	Extend CRD OpenAPI schemas to include `encryption` blocks.
deploy/.yaml, deploy/backup/.yaml	Update shipped CRDs/bundles and example manifests with `encryption` fields.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated 15 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Language not supported

Comments suppressed due to low confidence (2)

cmd/sidecar/handler/backup/create.go:111

This if block is a no-op and can be removed (the actual wait already happens earlier via awaitEncryptionKeyFile). Leaving empty conditionals makes the control flow harder to follow.

	xtrabackup.Env = envs(backupConf)

	xbOut, err := xtrabackup.StdoutPipe()

cmd/sidecar/handler/backup/create.go:247

The default --encrypt=AES256 flag is appended even when no encryption key is configured. That will cause previously-unencrypted backups to start failing (xtrabackup encryption requires a key or key file). Only set a default --encrypt= when encryption is actually enabled (e.g., when EncryptionKeyFile is set).

	}

	return args
}

func getClusterType() apiv1.ClusterType {

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Language not supported

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot

Pull request overview

Copilot reviewed 34 out of 35 changed files in this pull request and generated 7 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Language not supported

 	return ctrl.NewControllerManagedBy(mgr).
 		For(&apiv1.PerconaServerMySQL{}).
 		Watches(&corev1.Secret{}, enqueueClusterFromSecretsName(r.Client)).
+		Watches(&corev1.Secret{}, enqueueClusterFromEncryptionKeySecret(r.Client)).
 		Named("ps-controller").
 		Complete(r)


Copilot

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Language not supported

gkech · 2026-06-09T09:06:32Z

+            "encryptionKeySecret": {
+              "name": "${secretName}",
+              "key": "encryptionKey"


Asking since I didn't confirm. If we change the encryption key after the backup, and try to perform a restore, what key are we going to use? Are we sure that the restorer is not mounting the new key?

The restore fails in this case, since the original key is lost. If the user changes the encryption key, IMO they should take a new backup

egegunes

LGTM but please check copilot comments

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Language not supported

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot

Pull request overview

Copilot reviewed 52 out of 53 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Language not supported

JNKPercona · 2026-06-10T18:08:40Z

Test Name	Result	Time
async-ignore-annotations-8-4	passed	00:00:00
async-global-metadata-8-4	passed	00:14:47
async-upgrade-8-0	passed	00:00:00
async-upgrade-8-4	passed	00:13:02
auto-config-8-4	passed	00:00:00
config-8-4	passed	00:00:00
config-router-8-0	passed	00:00:00
config-router-8-4	passed	00:00:00
demand-backup-8-0	passed	00:00:00
demand-backup-8-4	passed	00:00:00
gr-pitr-minio-8-4	passed	00:00:00
gr-pitr-one-pod-8-4	passed	00:00:00
async-pitr-minio-8-4	passed	00:25:40
demand-backup-cloud-8-4	passed	00:22:04
demand-backup-retry-8-4	passed	00:00:00
demand-backup-incremental-8-0	passed	00:00:00
demand-backup-incremental-8-4	passed	00:00:00
async-data-at-rest-encryption-8-0	passed	00:00:00
async-data-at-rest-encryption-8-4	passed	00:00:00
gr-cross-cluster-8-0	passed	00:00:00
gr-cross-cluster-8-4	passed	00:00:00
gr-global-metadata-8-4	passed	00:14:20
gr-data-at-rest-encryption-8-0	passed	00:15:40
gr-data-at-rest-encryption-8-4	passed	00:00:00
gr-demand-backup-8-4	passed	00:00:00
gr-demand-backup-cloud-8-4	passed	00:00:00
gr-demand-backup-haproxy-8-4	passed	00:00:00
gr-demand-backup-incremental-8-0	passed	00:00:00
gr-demand-backup-incremental-8-4	passed	00:00:00
gr-finalizer-8-4	passed	00:00:00
gr-haproxy-8-0	passed	00:00:00
gr-haproxy-8-4	passed	00:00:00
gr-ignore-annotations-8-4	passed	00:00:00
gr-init-deploy-8-0	passed	00:00:00
gr-init-deploy-8-4	passed	00:00:00
gr-one-pod-8-4	passed	00:00:00
gr-recreate-8-4	passed	00:00:00
gr-scaling-8-4	passed	00:00:00
gr-scheduled-backup-8-4	passed	00:00:00
gr-scheduled-backup-incremental-8-4	passed	00:00:00
gr-security-context-8-4	passed	00:00:00
gr-self-healing-8-4	passed	00:00:00
gr-tls-cert-manager-8-4	passed	00:00:00
gr-users-8-4	passed	00:00:00
gr-upgrade-8-0	passed	00:00:00
gr-upgrade-8-4	passed	00:00:00
haproxy-8-0	passed	00:00:00
haproxy-8-4	passed	00:00:00
init-deploy-8-0	passed	00:00:00
init-deploy-8-4	passed	00:00:00
limits-8-4	passed	00:00:00
monitoring-8-4	passed	00:00:00
one-pod-8-0	passed	00:00:00
one-pod-8-4	passed	00:00:00
operator-self-healing-8-4	passed	00:00:00
pvc-resize-8-4	passed	00:00:00
recreate-8-4	passed	00:00:00
scaling-8-4	passed	00:00:00
scheduled-backup-8-0	passed	00:00:00
scheduled-backup-8-4	passed	00:00:00
scheduled-backup-incremental-8-0	passed	00:00:00
scheduled-backup-incremental-8-4	passed	00:00:00
service-per-pod-8-4	passed	00:00:00
sidecars-8-4	passed	00:00:00
smart-update-8-4	passed	00:00:00
storage-8-4	passed	00:00:00
telemetry-8-4	passed	00:00:00
tls-cert-manager-8-4	passed	00:00:00
users-8-0	passed	00:00:00
users-8-4	passed	00:00:00
version-service-8-4	passed	00:00:00

Summary	Value
Tests Run	71/71
Job Duration	00:50:45
Total Test Time	01:45:36

commit: a5f2884
image: perconalab/percona-server-mysql-operator:PR-1243-a5f2884a

Copilot AI review requested due to automatic review settings March 18, 2026 04:26

pull-request-size Bot added the size/L 100-499 lines label Mar 18, 2026

Copilot started reviewing on behalf of mayankshah1607 March 18, 2026 04:26 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

mayankshah1607 added 4 commits June 3, 2026 12:53

wip

1a95b30

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

implement backup & restore

a5bf759

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

manifests

6607df3

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

refactoring and cleanup

577cf9a

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

mayankshah1607 force-pushed the K8SPS-409 branch from 27b1d97 to 577cf9a Compare June 3, 2026 12:30

pull-request-size Bot added size/XL 500-999 lines and removed size/L 100-499 lines labels Jun 3, 2026

Merge branch 'main' into K8SPS-409

d8b1861

Copilot AI review requested due to automatic review settings June 4, 2026 08:47

Copilot started reviewing on behalf of mayankshah1607 June 4, 2026 08:47 View session

linting and unit tests

c37a343

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot AI reviewed Jun 4, 2026

View reviewed changes

mayankshah1607 added 3 commits June 4, 2026 15:03

copilot

1b601d3

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

added unit test

d9061fd

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

allow specifing encryption algorithm on restore

d382880

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot AI review requested due to automatic review settings June 4, 2026 09:57

Copilot started reviewing on behalf of mayankshah1607 June 4, 2026 09:57 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread pkg/mysql/mysql.go

Comment thread pkg/mysql/mysql.go

Comment thread pkg/controller/psbackup/controller.go

Comment thread pkg/controller/ps/controller.go

Comment thread pkg/xtrabackup/xtrabackup.go

Comment thread cmd/sidecar/handler/backup/create.go

mayankshah1607 added 4 commits June 4, 2026 16:02

address copilot

eef879c

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

bug fix

ed4ffe1

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

add e2e test

244992d

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Merge branch 'main' into K8SPS-409

9db038e

Copilot AI review requested due to automatic review settings June 4, 2026 10:33

Copilot started reviewing on behalf of mayankshah1607 June 4, 2026 10:34 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

mayankshah1607 added 2 commits June 9, 2026 14:13

Merge branch 'main' into K8SPS-409

c793f56

Merge branch 'main' into K8SPS-409

d0f742f

Copilot AI review requested due to automatic review settings June 9, 2026 08:45

github-actions Bot added tests build labels Jun 9, 2026

Copilot started reviewing on behalf of mayankshah1607 June 9, 2026 08:45 View session

mayankshah1607 requested a review from egegunes June 9, 2026 08:49

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Comment thread pkg/mysql/mysql.go

Comment thread pkg/mysql/mysql.go

Comment thread pkg/controller/ps/controller.go Outdated

gkech reviewed Jun 9, 2026

View reviewed changes

egegunes previously approved these changes Jun 10, 2026

View reviewed changes

copilot

5bc9afd

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

mayankshah1607 dismissed egegunes’s stale review via 5bc9afd June 10, 2026 06:32

mayankshah1607 requested a review from egegunes June 10, 2026 06:33

typo

ca07b3c

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot AI review requested due to automatic review settings June 10, 2026 07:20

Copilot started reviewing on behalf of mayankshah1607 June 10, 2026 07:20 View session

gkech previously approved these changes Jun 10, 2026

View reviewed changes

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Comment thread pkg/mysql/mysql.go

Comment thread pkg/mysql/mysql.go

Comment thread pkg/controller/ps/controller.go

Comment thread pkg/controller/ps/controller.go Outdated

address copilot optimizations

bf9b95d

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

mayankshah1607 dismissed gkech’s stale review via bf9b95d June 10, 2026 09:27

remove APIReader

9b5207d

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot AI review requested due to automatic review settings June 10, 2026 14:04

Copilot started reviewing on behalf of mayankshah1607 June 10, 2026 14:04 View session

rename version file to .version

a5f2884

Signed-off-by: Mayank Shah <mayank.shah@percona.com>

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Comment thread pkg/mysql/mysql.go

Comment thread pkg/mysql/mysql.go

Comment thread cmd/sidecar/handler/backup/create.go

Comment thread pkg/controller/psbackup/controller.go

egegunes approved these changes Jun 11, 2026

View reviewed changes

gkech approved these changes Jun 11, 2026

View reviewed changes

mayankshah1607 merged commit bf6828f into main Jun 11, 2026
16 checks passed

mayankshah1607 deleted the K8SPS-409 branch June 11, 2026 07:34

Conversation

mayankshah1607 commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CHANGE DESCRIPTION

CHECKLIST

Uh oh!

it-percona-cla commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gkech Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

mayankshah1607 Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

egegunes left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mayankshah1607 commented Mar 18, 2026 •

edited

Loading

it-percona-cla commented Mar 18, 2026 •

edited

Loading